Explicit Max Margin Input Feature Selection for Nonlinear SVM using Second Order Methods

Incorporating feature selection in nonlinear SVMs leads to a large and challenging nonconvex minimization problem, which can be prone to suboptimal solutions. We use a second order optimization method that utilizes eigenvalue information and is less likely to get stuck at suboptimal solutions. We devise an alternating optimization approach to tackle the problem efficiently, breaking it down into a convex subproblem, corresponding to SVM optimization, and a nonconvex subproblem for feature selection. Importantly, we show that a naive approach, which implicitly maximizes the margin, can be susceptible to saddle point solutions. We propose a new explicit margin maximization method, which overcomes the drawbacks of the implicit approach. To avoid being trapped at saddle points, we first introduce an auxiliary variable and perform alternating optimization in the extended space, and then subsequently, to improve solution quality, we relax the use of geometric margin and maximize the functional margin in the feature selection subproblem. We demonstrate how this technique can also be applied to 1-norm linear programming SVMs. Experiment results show the explicit margin approach works effectively in the presence of class noise and many irrelevant features, consistently outperforming leading filter, wrapper, and other embedded nonlinear feature selection methods.